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Distribution  of  Attention,  Situation  Awareness,  and  Workload  in  a 
Passive  Air  Traffic  Control  Task:  Implications 
FOR  Operational  Errors  and  Automation 


INTRODUCTION 

In  the  history  of  the  Federal  Aviation  Administra¬ 
tion  (FAA),  no  aircraft  have  collided  while  under 
positive  control  in  en  route  airspace.  However,  air¬ 
craft  have  violated  prescribed  separation  minima  and 
approached  in  close  proximity.  This  event  can  occur 
as  a  result  of  either  a  pilot  deviation  from  clearances  or 
an  operational  error  (OE)  by  an  air  traffic  controller. 
An  OE  takes  place  when  the  controller  allows  less  than 
applicable  minimum  separation  between  an  aircraft, 
another  aircraft,  or  an  obstruction.  Standards  for 
separation  minima  are  described  in  the  Air  Traffic 
Control  (ATC)  Handbook  (FAA  Order  7110.65J) 
and  supplemental  instructions.  While  there  is  consid¬ 
erable  complexity  in  those  standards,  the  criteria  are 
established  as  2,000  feet  of  vertical  separation  or  5 
miles  of  horizontal  separation  between  aircraft  oper¬ 
ating  at  altitudes  between  29,000  feet  and  45,000  feet. 
For  aircraft  operating  under  instrument  flight  rules 
(IFR)  at  flight  levels  below  29,000  feet,  a  minimum  of 
1,000  feet  of  vertical  separation,  or  5  miles  of  horizon¬ 
tal  separation  are  required.  These  separation  stan¬ 
dards  provide  tolerance  zones  ensuring  that  aircraft 
pass  well  clear  of  one  another. 

A  relatively  small  number  of  OEs  occur  nationwide 
each  year.  In  1 993, 430  OEs  were  recorded  at  en  route 
air  traffic  control  centers  in  the  US,  with  37,170,000 
aircraft  handled.  In  an  effort  to  ensure  flight  safety, 
there  is  a  desire  to  reduce  the  number  of  OEs  that 
occur.  Doing  so  requires  an  understanding  of  why 
these  errors  occur  and  the  factors  that  are  likely  to 
increase  the  probability  of  an  operational  error. 

Rodgers  and  Nye  (1993)  investigated  causal  factors 
associated  with  (minor  and  moderate)  OEs  occurring 
at  en  route  ATC  facilities  over  a  three  and  one-half 
year  period,  based  on  the  FAA’s  Operational  Error 
Data  Base.  This  data  base  records  circumstances  asso¬ 
ciated  with  OEs,  as  identified  by  quality  assurance 


(QA)  investigators  following  the  OEs'  occurrence. 
They  found  that  36%  of  OEs  involved  problems  with 
communications  (including  20%  that  were  specifi¬ 
cally  readback  errors),  15%  involved  coordination 
problems,  3%  involved  deficiencies  in  position  relief 
briefings,  13%  were  associated  with  problems  in  data 
posting,  and  59%  were  related  to  the  radar  display 
(including  14%  that  involved  misidentification  of 
information  and  47%  that  involved  inappropriate  use 
of  displayed  data).  Some  errors  are  attributed  to 
multiple  causal  factors. 

A  number  of  research  studies  have  sought  to  inves¬ 
tigate  the  relationship  between  estimates  of  controller 
workload  and  the  incidence  of  OEs,  Operational 
errors  have  been  found  to  occur  under  both  high  and 
low  workload  conditions,  with  more  errors  occurring 
under  low  and  moderate  levels  of  workload  than 
under  high  levels  of  workload  (Kinney,  Spahn,  & 
Amato,  1977;  Schroeder,  1982;  Stager  &  Hameluck, 
1 990) .  It  is  unclear  from  these  data,  however,  whether 
this  reflects  a  decreased  tendency  to  make  errors  under 
high  workload  conditions  or  a  lower  frequency  of  high 
workload  conditions  occurring  overall. 

A  recent  study  by  Schroeder  and  Nye  (1993)  found 
a  positive  correlation  between  the  number  of  aircraft 
under  the  Air  Traffic  Control  Specialist's  (ATCS's) 
control  (normalized  for  the  average  number  of  aircraft 
per  ATCS  in  that  center)  and  the  occurrence  of  OEs 
involving  data  posting,  position  relief  briefings,  and 
misuse  of  displayed  radar  data.  They  also  found  an 
association  between  OEs  involving  coordination  prob¬ 
lems  and  both  a  lower  than  average  number  of  aircraft 
and  a  higher  than  average  number  of  aircraft.  There 
was  no  association  found  between  number  of  aircraft 
and  OEs  involving  communications  problems. 

It  should  be  noted  that  these  studies  rely  on  esti¬ 
mates  of  controller  workload  that  were  made  follow¬ 
ing  the  OE  by  FAA  QA  investigators  who  used  a 
simple  workload  scale  (1  to  5,  non-anchored)  that  is 
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not  clearly  defined.  The  investigators  typically  receive 
little  or  no  training  on  what  factors  to  include  in  their 
workload  estimates.  Thus,  non-standardization,  as 
well  as  potential  inaccuracy  of  estimates  made  after 
the  fact  are  limitations  affecting  the  workload  mea¬ 
sures  used  in  these  studies. 

There  has  also  been  an  interest  in  determining  to 
what  degree  OEs  involve  a  problem  with  controller 
situation  awareness  (SA)  —  their  mental  picture  of  the 
constantly  changing  air  traffic  situation.  Formally 
defined,  SA  is  the  “perception  of  the  elements  in  the 
environment  within  a  volume  of  time  and  space,  the 
comprehension  of  their  meaning,  and  the  projection 
of  their  status  in  the  near  future”  (Endsley,  1988).  It 
encompasses  not  only  an  awareness  of  specific  key 
elements  in  the  situation  (Level  1  SA),  but  also  an 
integration  and  comprehension  of  that  information 
in  light  of  operational  goals  (Level  2  SA),  along  with 
an  ability  to  project  future  states  of  the  system  (Level 
3  SA).  These  higher  levels  of  SA  (Levels  2  and  3)  are 
felt  to  be  particularly  critical  for  effective  functioning 
in  complex  environments,  such  as  air  traffic  control. 
In  air  traffic  control,  SA  involves  the  continuous  aware¬ 
ness  of  the  location  of  each  aircraft,  along  with  pertinent 
aircraft  parameters  (speed,  heading,  communications, 
etc.)  and  their  projected  future  locations  relative  to 
each  other,  among  many  other  pieces  of  information. 


to  provide  minimum  separation  and  efficient  aircraft 
movement.  A  complete  delineation  of  SA  information 
requirements  for  en  route  ATC  is  provided  in  Endsley 
and  Rodgers  (1994). 

Endsley  (1995a)  developed  a  taxonomy  of  SA  er¬ 
rors,  describing  causal  factors  associated  with  the 
occurrence  of  SA  errors  (shown  in  Table  1).  This 
study  found  that  88%  of  major  air  carrier  accidents 
associated  with  pilot  error  involved  a  problem  with 
situation  awareness.  Of  these,  72%  involved  prob¬ 
lems  with  Level  1  SA  ,  22%  involved  problems  with 
Level  2  SA,  and  6%  involved  problems  with  Level  3  SA. 

Table  1  lists  general  causal  factors  in  the  SA  error 
taxonomy  associated  with  a  lack  of  situation  aware¬ 
ness  at  each  of  its  three  levels.  Problems  with  SA  at 
Level  1  (perception  of  the  elements  in  the  environ¬ 
ment)  can  occur  when  needed  information  (1)  is  not 
available  through  a  failure  of  the  system  design  or 
communications  process,  (2)  is  available  but  is  diffi¬ 
cult  to  detect  or  perceive,  (3)  is  not  observed  or 
monitored,  often  due  to  distractions,  attentional  nar¬ 
rowing,  high  taskload,  or  failures  in  the  scanning 
process,  (4)  is  misperceived,  frequently  due  to  errone¬ 
ous  expectations,  or  (5)  is  initially  perceived,  but  is 
forgotten  due  to  failures  in  working  memory,  and 
thus  is  not  taken  into  account. 


Table  1.  SA  Error  Taxonomy  (modified  from  Endsley,  1995a) 

Level  1 :  Failure  to  correctly  perceive  information 

•  Data  not  available 

•  Data  difficult  to  detect  or  perceive 

•  Failure  to  monitor  or  observe  data 

•  Misperception  of  data 

•  Memory  failure 

Level  2:  Failure  to  comprehend  situation 

•  Lack  of  or  poor  mental  model 

•  Use  of  incorrect  mental  model 

•  Over-reliance  on  default  values  in  mental  model 

•  Other 

Level  3:  Failure  to  project  situation  into  the  future 

•  Lack  of  or  poor  mental  model 

•  Overprojection  of  current  trends 

•  Other 
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Problems  with  SA  at  Level  2  (comprehension  of  the 
situation)  can  occur  when  information  is  correctly 
perceived,  but  its  significance  or  meaning  is  not  com¬ 
prehended.  This  may  be  because  (1)  a  good  mental 
model  for  combining  and  processing  perceived  infor¬ 
mation  is  not  available,  (2)  an  incorrect  mental  model 
is  selected,  leading  the  person  to  improperly  interpret 
perceived  information,  (3)  there  is  an  overreliance  on 
default  values  (general  expectations  about  how  parts 
of  the  system  will  function)  in  the  mental  model,  or 
(4)  other  factors,  such  as  limited  working  memory 
with  which  to  process  information  or  lapses  in  usual 
cognitive  processes. 

Problems  with  SA  at  Level  3  (projection  of  future 
behavior  of  elements  in  the  environment)  may  occur 
if  a  person  understands  what  is  going  on  in  the  current 
situation  but  has  trouble  projecting  what  that  means 
for  the  future.  This  may  occur  because  (1)  a  good 
mental  model  that  provides  for  predictions  of  system 
dynamics  and  behavior  is  not  available,  (2)  there  is  a 
tendency  to  project  future  behavior  as  a  linear  func¬ 
tion  of  current  systems  dynamics  when  the  dynamics 
may  change  in  non-linear  ways,  or  (3)  other  lapses  in 
cognitive  processing. 

The  SA  error  taxonomy  was  applied  in  a  recent 
study  of  146  incidents  involving  reported  problems  in 
SA  among  both  pilots  and  controllers  in  NASA’s 
voluntary  Aviation  Safety  Reporting  System  (ASRS) 
(Jones  &C  Endsley,  1996).  (Only  incidents  involving 
air  traffic  control  are  discussed  here.)  Of  the  33 
incidents  involving  air  traffic  controllers,  69%  in¬ 
volved  problems  with  Level  1  SA,  19%  involved 
problems  with  Level  2  SA,  and  12%  involved  prob¬ 
lems  with  Level  3  SA.  Of  the  Level  1  SA  errors,  the 
most  common  problem  was  a  failure  to  monitor  or 
observe  data  (51.5%).  This  was  most  frequently  due 
to  task  distraction  (53%  of  these  cases),  followed  by 
problems  with  high  workload  (17.6%),  vigilance 
(11.8%),  and  other  miscellaneous  causes  (17.7%), 
such  as  a  failure  to  scan  the  runway,  failure  to  notice 
an  aircraft  overshoot,  and  failure  to  notice  traffic  on 
the  runway.  Other  causal  factors  were  also  related  to 
Level  1  S A  errors:  18.2%  involved  cases  where  needed 
data  were  not  available,  18.2%  involved  cases  where 
controllers  forgot  important  information  (frequently 
under  high  workload),  6.1%  involved  data  that  were 


hard  to  discriminate  or  detect,  and  6.1%  involved  the 
misperception  of  information. 

Level  2  SA  errors  were  attributed  to  an  incomplete 
or  inaccurate  mental  model  (22.2%),  the  use  of  an 
incorrect  mental  model  (22.2%),  over- reliance  on 
default  values  (22.2%),  and  other  miscellaneous  fac¬ 
tors  (33.3%).  Level  3  SA  errors  were  attributed  to 
over-projection  of  current  trends  (33.3%)  and  other 
miscellaneous  factors  (66.7%).  There  were  no  obvi¬ 
ous  cases  of  Level  3  errors  due  to  poor  mental  models. 

It  should  be  noted  that  the  errors  by  air  traffic 
controllers  examined  in  the  Jones  and  Endsley  study 
involved  voluntarily  reported  information  from  a 
variety  of  ATC  facilities  including  air  route  traffic 
control  centers  (ARTCC),  TRACONs,  and  towers 
(both  local  and  ground  control).  As  such,  while  the 
data  do  provide  some  information  about  the  types  of 
errors  that  may  occur  across  different  types  of  ATC, 
this  cannot  be  viewed  as  a  truly  representative,  or 
completely  unbiased,  sample  of  controller  errors.  In 
addition,  it  is  often  difficult  to  ascertain  exactly  why 
some  of  the  errors  occurred  from  the  limited  informa¬ 
tion  available  in  such  reports. 

Most  information  about  errors  is  based  on  the 
analysis  of  available  historical  reports.  These  reports 
are  often  not  developed  with  the  objective  of  examin¬ 
ing  detailed  causal  factors,  are  usually  based  on  after- 
the-fact  interviews  that  may  be  incomplete  or  biased, 
and  frequently  suffer  from  problems  of  inconsistency, 
as  different  people  usually  conduct  each  investiga¬ 
tion.  The  objective  of  the  present  study  was  to  collect 
more  detailed  data  about  OEs  than  what  is  available  in 
such  accounts,  providing  for  a  better  understanding 
of  factors  that  may  contribute  to  their  occurrence. 
This  study  focused  on  data  gathered  on  OEs  from  the 
Atlanta  ARTCC. 

To  gain  more  insight  into  the  nature  of  OEs,  the 
Systematic  Air  Traffic  Operations  Research  Initiative 
(SATORI)  system  was  developed  (Rodgers  &  Duke, 
1993).  SATORI  graphically  recreates  a  visual  display 
of  the  radar  data  recorded  during  actual  air  traffic 
control  (based  on  computer  tapes  routinely  recorded 
at  each  air  traffic  control  facility),  synchronized  with 
the  recorded  audio  tapes  of  communications  between 
controllers  and  between  controllers  and  pilots.  Prior 
to  the  development  of  the  SATORI  system,  it  was  not 
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possible  for  the  FAA  QA  team  investigating  errors  to 
review  the  control  situation  in  a  format  like  the  one 
presented  to  the  controller  when  the  OE  occurred. 
That  is,  the  dynamics  of  the  situation  (the  interaction 
between  control  actions  and  displayed  data)  were 
unavailable  for  review,  not  only  by  the  QA  team 
investigating  the  irregularity,  but  also  by  the  control¬ 
ler  who  committed  the  error.  This  limited  not  only 
the  extent  to  which  reliable  and  accurate  determina¬ 
tions  of  causal  factors  could  be  made  for  an  error,  but 
also  the  extent  to  which  the  effects  of  the  dynamic 
situation  on  controller  SA  could  be  determined.  The 
SATORI  system  allows  QA  specialists  and  controllers 
to  view  an  accurate,  dynamic  representation  of  the 
ATC  data  associated  with  an  OE. 

In  the  present  study,  the  re-creation  of  OEs  using 
SATORI  was  combined  with  a  modification  of  the 
Situation  Awareness  Global  Assessment  Technique 
(SAGAT)  (Endsley,  1988).  SAGAT  is  a  technique 
used  during  simulations  in  which  the  simulation  is 
frozen  at  random,  unexpected  intervals  with  all  dis¬ 
play  screens  blanked  and  the  operator  of  the  simula¬ 
tion  is  queried  about  the  state  of  the  current  situation. 
The  operator’s  perceptions  are  then  compared  to  the 
actual  state  of  the  environment  to  provide  an  objective 
assessment  of  the  operator’s  situation  awareness.  The 
use  of  SAGAT  to  measure  situation  awareness  in 
aircraft  simulations  has  been  extensively  validated 
(Endsley,  1990a;  1990b). 

In  this  study,  SAGAT  was  modified  to  include 
queries  that  pertain  to  major  factors  associated  with 
SA  in  en  route  ATC,  based  on  an  analysis  by  Endsley 
and  Rodgers  (1994).  As  a  second  modification,  the 
technique  was  employed  in  conjunction  with  SATORI, 
which  involves  the  passive  viewing  of  a  situation,  as 
opposed  to  an  interactive  simulation  in  which  the 
subject  is  involved.  While  it  is  not  clear  how  the  SA  of 
a  passive  observer  differs  from  that  of  an  active  partici¬ 
pant,  this  measure  should  still  provide  an  indication 
of  the  way  in  which  controllers  distribute  their  atten¬ 
tion  to  various  factors  involved  in  the  scenarios.  As 
this  study  involves  currently  certified  controllers  view¬ 
ing  re-creations  of  real  OEs,  the  combined  use  of 
SATORI  and  SAGAT  may  provide  unique  insight 
into  factors  affecting  OEs  in  operational  settings.  Due 
to  their  rare  occurrence  and  the  limited  conditions 


usually  involved  in  simulations,  observing  OEs  in 
simulations  can  prove  to  be  quite  difficult. 

In  addition,  it  may  be  considered  that  SA  under 
passive  viewing  conditions  may  be  analogous  to  actual 
controller  tasks  if  the  ATC  system  ever  becomes 
highly  automated.  While  many  of  the  tasks  a  control¬ 
ler  currently  performs  support  the  acquisition  and 
maintenance  of  SA,  the  task  environment  under  con¬ 
ditions  of  high  automation  may  change  such  that  the 
requirement  to  perform  many  routine  tasks  is  elimi¬ 
nated.  In  this  case,  the  controller  would  become  a 
monitor  of  the  air  traffic  situation,  which  would  be 
controlled  by  an  automated  system.  Since  the  sce¬ 
narios  recreated  for  subjects  using  SATORI  similarly 
involve  the  passive  monitoring  of  a  situation  that  is 
actually  controlled  by  another,  this  study  may  provide 
some  insights  into  SA  with  a  hypothetical  highly 
automated  system.  This  is  a  concern,  as  there  is  some 
indication  that  SA  may  be  compromised  under  highly 
automated  systems  (Endsley  &  Kiris,  1995). 

METHOD 

Subjects 

Twenty  volunteer  subjects  participated  in  the  study. 
All  were  experienced,  full  performance  level  (FPL)  air 
traffic  control  specialists  at  Atlanta  ARTCC.  The  20 
subjects  included  4  subjects  viewing  OEs  in  each  of  5 
areas  of  specialization  in  the  facility,  as  shown  in 
Figure  1.  All  subjects  were  certified  in  the  area  of 
specialization  for  the  re-created  errors  that  they  ob¬ 
served  during  the  study.  Subjects  were  relieved  from 
their  duties  on  the  air  traffic  control  room  floor  to 
participate  in  the  study.  Once  subjects  completed 
their  participation,  they  returned  to  their  assigned 
duties. 

Procedure 

Fifteen  OEs  that  occurred  in  the  Atlanta  Air  Route 
Traffic  Control  Center  (ARTCC)  in  1993  and  1994 
were  recreated  using  SATORI.  These  errors  were 
selected  from  errors  involving  a  single  ATC  sector, 
based  on  the  availability  of  complete  and  legible 
computer  data  tapes  and  audio  recordings.  Three 
errors  in  each  of  five  areas  of  specialization  of  the 
center  were  selected.  (One  error  was  eliminated  during 
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Figure  1 .  Experimental  Design 


testing  due  to  problems  with  the  data  tapes,  leaving  14 
errors  to  be  included  in  the  analysis.) 

Subjects  were  provided  with  a  set  of  instructions 
and  signed  a  voluntary  subject  consent  form.  They 
were  then  shown  scenarios  involving  three  errors  from 
sectors  in  the  area  of  specialization  on  which  they  were 
certified.  Each  scenario  consisted  of  a  re-creation  of 
the  1 0  minutes  immediately  prior  to  the  occurrence  of 
the  OE.  Twice  during  each  scenario,  the  re-creation 
was  halted  and  the  screen  blanked.  The  first  freeze 
occurred  two  minutes  prior  to  the  occurrence  of  the 
error  and  the  second  freeze  occurred  at  the  time  of  the 
OE  in  each  scenario.  Although  subjects  were  in¬ 
formed  that  freezes  would  occur,  they  were  not  in¬ 
formed  of  the  timing  of  the  freezes  or  the  occurrence 
of  the  error. 

During  each  freeze,  subjects  were  provided  with  a 
map  of  the  sector.  Sector  boundaries,  navigation  aids, 
airways  and  intersection  markings  were  shown  on  the 
map;  however,  no  aircraft  were  included.  Subjects 
were  asked  to  indicate  the  location  of  all  known 


aircraft  on  the  map,  and,  for  each  aircraft,  to  indicate 
or  make  a  judgment  of: 

(1)  if  the  aircraft  were: 

(a)  in  the  displayed  sector’s  control, 

(b)  other  aircraft  in  the  sector  not  under  sector 
control,  or 

(c)  would  be  in  the  sector’s  control  in  the  next 
two  minutes, 

(2)  aircraft  call  sign, 

(3)  aircraft  altitude, 

(4)  aircraft  groundspeed, 

(5)  aircraft  heading, 

(6)  the  next  sector  the  aircraft  would  transition  to, 

(7)  whether  the  aircraft  was  climbing,  descending 
or  level, 

(8)  whether  the  aircraft  was  in  a  right  turn,  left  turn 
or  straight, 

(9)  which  pairs  of  aircraft  had  lost,  or  would  lose 
separation  if  they  stayed  on  their  current  (as¬ 
signed)  courses, 

(10)  which  aircraft  would  be  leaving  the  sector  in  the 
next  two  minutes, 
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(11)  which  aircraft  had  received  clearances  that  had 
not  been  completed  and,  for  those,  whether  the 
aircraft  received  its  clearance  correctly  and 
whether  the  aircraft  was  conforming  to  its  clear¬ 
ance,  and 

(12)  which  aircraft  were  currently  being  impacted  by 
weather  or  would  be  impacted  in  the  next  five 
minutes. 

Of  these,  queries  1,  2,  3,  4,  5,  7,  and  8  can  be 
regarded  as  pertaining  to  Level  1  SA,  and  queries  6,  9, 
10,  1 1,  and  12  can  be  regarded  as  pertaining  to  Levels 
2  and  3  SA. 

Following  the  completion  of  the  questionnaire, 
each  subject  completed  a  NASA-TLX  subjective  work¬ 
load  rating  (Hart  &  Staveland,  1988),  indicating  the 
amount  of  workload  they  felt  they  would  be  under  if 
they  were  controlling  the  traffic  in  the  scenario  pre¬ 
sented.  Following  completion  of  the  NASA-TLX  ques¬ 
tionnaire,  the  scenario  was  resumed  until  the  second 
freeze.  Then,  the  SA  queries  and  NASA-TLX  ques¬ 
tionnaire  were  again  presented  in  the  same  order, 
following  which  the  scenario  was  terminated  and  the 
next  scenario  presented.  After  all  three  scenarios  had 
been  presented,  subjects  completed  a  NASA-TLX 
workload  paired  comparison  ranking  form,  allowing 
each  subject’s  ratings  on  each  NASA-TLX  sub¬ 
dimension  to  be  weighted  based  on  the  subjective 
importance  of  the  subdimension  to  each  subject. 

Apparatus 

SATORI  re-creations  were  presented  on  a  DEC 
3000-300  Alpha  computer  system  using  dual  Sony 
19'inch  high-resolution  (1280  x  1024)  color  moni¬ 
tors.  NASA-TLX  ratings  were  obtained  using 
HyperCard  on  a  Macintosh  Powerbook. 

RESULTS  AND  DISCUSSION 

SA  Questionnaire 

Subjects’  responses  to  each  question  were  scored 
for  accuracy  based  on  computer  data  for  each  aircraft 
at  the  time  of  each  freeze.  Subjects’  indications  of  each 
aircraft’s  location  on  the  map  were  matched  to  the 
closest  aircraft  actually  present  in  the  sector  at  the 
time  of  the  freeze  and  the  distance  error  recorded.  The 
percentage  of  aircraft  present  that  were  reported  by 


the  subject  was  calculated.  Following  that,  scoring  for 
each  subsequent  question  was  calculated  as  the  num¬ 
ber  of  correct  responses  compared  to  the  number  of 
aircraft  that  the  subject  reported  knowing  about  (e.g., 
percent  correct  for  altitude  was  calculated  as  the 
number  of  correct  aircraft  altitudes  reported,  divided 
by  the  total  number  of  aircraft  reported.)  Subjects’ 
responses  for  each  question  were  scored  as  either 
correct  or  incorrect,  based  on  operationally  deter¬ 
mined  tolerance  intervals  (as  listed  in  Table  2).  Miss¬ 
ing  responses  were  scored  as  incorrect.  It  should  be 
noted  that  the  selected  sample  of  errors  had  a  higher 
number  of  aircraft  (Mean  =  12.9,  S.D.  =  5.6)  than 
typically  occur  at  ZTL  (Mean  =  8.43,  S.D.  =  1.86); 
however,  given  the  focus  of  the  study  on  controller 
situation  awareness  in  a  future  automated  ATC  sys¬ 
tem,  it  was  deemed  appropriate.  Automation  and 
higher  traffic  levels  have  been  associated  with  the 
future  operations  of  NAS.  The  study  employed  a 
passive  ATC  task  involving  situation  monitoring, 
although  this  task  is  quite  different  than  the  current 
ATC  task,  which  involves  active  control,  it  generally 
represents  the  cognitive  task  environment  that  might 
be  associated  with  excessive  automation. 

Means  and  standard  deviations  for  subject  response 
accuracy  are  shown  in  Table  2.  On  average,  12.8 
aircraft  were  present  at  the  time  of  the  freezes  (range 
4  to  23)  across  scenarios.  Of  these,  subjects  on  average 
reported  8.0  aircraft  or  67.1%  of  the  aircraft  present. 
Mean  distance  error  was  9.6  miles  (.68  inches)  from 
the  aircrafts’  reported  location  to  their  actual  loca¬ 
tion.  This  may  reflect  aircraft  movement  occurring 
during  the  visual  scan  (while  the  controller  was  view¬ 
ing  other  aircraft)  or  may  also  be  an  artifact  of  the 
passive  viewing  procedure  used  in  this  study. 

For  the  aircraft  reported,  the  correctness  of  subject 
responses  on  the  remaining  questions  was  calculated. 
Subjects  correctly  identified  the  control  level  of  the 
aircraft  (in  sector  control,  other  aircraft  in  sector,  will 
be  in  sector  control  in  the  next  2  minutes)  for  73.8% 
of  the  aircraft  reported.  Aircraft  callsigns  were  often 
incomplete.  The  initial  alphabetical  part  of  the  callsign 
(indicating  airline  company,  military,  or  civil  aircraft 
designation)  was  reported  correctly  73.8%  of  the 
time.  The  numerical  part  of  the  callsign  (the  aircraft 
identification  number)  was  reported  correctly  for  only 
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Table  2.  Awareness  of  Situation  Across  all  Subjects  and  Scenarios 


Variable _ 

Actual  aircraft  present  (number) 

Aircraft  reported  (%) 

Distance  error  (miles) 

Control  level  (%  correct) 

Call  sign;  alphabetic  (%  correct) 

Call  sign:  numeric  (%  correct) 

Altitude  (+/-  300  feet)  (%  correct) 

Change  in  altitude  (%  correct) 

Speed  (+/- 1 0  knots)  (%  correct) 

Heading  (+/- 15  degrees)  (%  correct) 

Turn  (%  correct) 

Separation  problems  (%  correct) 

Transition  to  next  sector  (%  correct) 

Assigned  clearances  complete  (%  correct) 
Assigned  clearance  correct  (%  correct) 
Assigned  clearance  conformance  (%  correct) 
Weather  impact  (%  correct) 


Mean 

Std.  Di 

12.9 

5.6 

67.1 

18.0 

9.6 

4.5 

73.8 

17.3 

79.9 

23.4 

38.4 

32.0 

59.7 

22.1 

66.4 

25.6 

28.0 

25.6 

48.4 

30.6 

35.1 

40.2 

86.2 

32.3 

63.5 

45.1 

23.2 

22.9 

74.4 

43.9 

82.9 

37.9 

60.7 

49.1 

38.4%  of  the  aircraft.  It  should  be  noted  that  other 
studies  have  found  that,  in  general,  4%  of  OEs  involve 
readback  errors  associated  with  aircraft  identification 
(Rodgers  &  Nye,  1993.  The  low  level  of  accuracy  in 
recall  knowledge  of  aircraft  callsigns  is  probably  highly 
indicative  of  these  readback  errors,  as  it  indicates  that 
controllers  may  not  attend  to  or  retain  much  informa¬ 
tion  on  aircraft  callsign  in  working  memory,  particu¬ 
larly  the  identification  number. 

Aircraft  altitude  was  correctly  reported  (±  300  ft) 
for  59.7%  of  the  aircraft  (mean  error  of  655  ft).  The 
aircraft  were  correctly  identified  as  ascending,  de¬ 
scending,  or  level  66.4%  of  the  time.  Correct  ground- 
speed  (±10  knots)  was  reported  for  only  28.0%  of  the 
aircraft  (mean  error  21.8  knots).  Correct  aircraft  head¬ 
ing  (±15  degrees)  was  reported  for  48.4%  of  the 
aircraft  (mean  error  15.6  degrees) .  Only  35 . 1  %  of  the 
aircraft  were  correctly  identified  as  being  in  a  left  turn, 
right  turn,  or  proceeding  straight  ahead.  These  results 
indicate  that  subjects  were  fairly  poor  at  keeping  up 
with  the  dynamics  of  the  aircraft  in  the  scenario,  at 
least  for  many  of  the  aircraft. 

An  argument  can  be  made  that  perhaps  subjects 
simply  did  not  retain  this  type  of  detailed  information 
about  each  aircraft  (Level  1  SA)  and  instead,  main¬ 
tained  awareness  of  higher  level  situation  comprehen¬ 


sion  and  projection  issues  (e.g.,  aircraft  separation 
and  future  projections  of  actions).  Previous  research, 
however,  indicates  that  people  do  maintain  task  rel¬ 
evant  information  about  Level  1  SA  elements  that  can 
be  recalled  reliably  under  SAGAT  testing  when  ac¬ 
tively  performing  in  a  simulation  (Endsley,  1990a). 
There  is  also  evidence  that  this  measurement  tech¬ 
nique  is  reflective  of  subject  attention  allocation  across 
sources  of  information  (Fracker,  1990.  It  is  more 
likely,  therefore,  that  these  measures  do  provide  some 
indication  of  the  v^ays  in  which  subjects  in  this  study 
were  deploying  their  attention  across  displayed  infor¬ 
mation,  at  least  on  a  relative  basis. 

The  subjects'  higher  level  of  understanding  of  the 
scenarios  was  also  evaluated.  The  aircraft  pairs  they 
identified  as  having  “lost  or  will  lose  separation  if  they 
stay  on  their  current  (assigned)  courses”  was  com¬ 
pared  to  those  aircraft  that  actually  had  lost  or  would 
lose  separation  (in  the  following  two  minutes)  at  the 
time  of  the  freeze.  Subjects  correctly  identified  86.2% 
of  these  aircraft  pairs.  (Aircraft  pairs  that  the  subject 
identified  as  having  potential  separation  problems, 
but  did  not,  were  not  scored.)  Subjects  correctly 
identified  63.5%  of  aircraft  that  would  be  leaving  the 
sector  in  the  next  two  minutes.  Thus,  they  did  not  appear 
to  be  fully  aware  of  upcoming  sector  transitions. 
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Subjects  correctly  identified  only  23.2%  of  aircraft 
that  had  not  yet  completed  control  assignments.  Of 
those  that  they  identified  as  not  having  completed  an 
assignment,  subjects  correctly  determined  in  74.4% 
of  the  cases  if  the  aircraft  had  correctly  received  its 
assignment  and  correct  in  82.9%  of  the  cases  if  the 
aircraft  was  conforming  to  its  assignment.  Overall, 
subjects  did  not  attend  well  to  an  aircraft  after  a 
clearance  was  given,  in  terms  of  monitoring  for  com¬ 
pliance  or  progress  in  completing  the  control  action, 
most  likely  because  they  may  have  been  concentrating 
on  other  traffic  present. 

Subjects  were  incorrect  in  identifying  weather  as  a 
current  impact  (or  impact  in  the  next  5  minutes)  in 
39.3%  of  the  scenarios.  This  is  perplexing,  in  that 
even  though  light  and  heavy  weather  symbols  were 
displayed  in  some  scenarios,  poor  weather  did  not 
impact  traffic  in  any  of  the  scenarios  presented.  This 
finding  most  likely  indicates  that  controllers  have 
difficulty  estimating  the  impact  of  weather  on  air 
traffic  based  on  available  data,  an  issue  which  has 
previously  been  raised  by  controllers. 

The  frequency  of  correct  responses  on  each  variable 
provides  some  insight  into  the  trade-offs  that  control¬ 
lers  make  in  allocating  their  limited  attention  across 
multiple  aircraft  and  pieces  of  information  that  com¬ 
pete  for  that  attention.  This  analysis  is  not  meant  to  be 
critical  regarding  the  information  controllers  did  not 
attend  to  or  retain  in  working  memory.  Attention 
allocation  strategies,  such  as  those  indicated  here,  are 
needed  and  are  effective  most  of  the  time  in  dealing 
with  the  demands  of  controlling  air  traffic,  as  can  be 
demonstrated  by  the  effective  daily  performance  of 
controllers  and  relatively  low  nationwide  error  rates. 
A  point  that  can  be  made,  however,  is  that  these 
strategies  may  lead  to  a  lack  of  situation  awareness  that 
occasionally  (due  to  a  probabilistic  link  between  SA 
and  performance  [Endsley,  1995b])  results  in  errors. 
This  point  is  reinforced  in  that  the  patterns  of  atten¬ 
tion  demonstrated  here  can  be  correlated  with  certain 
systematic  characteristics  of  OEs. 

It  should  also  be  noted  that  a  fairly  high  degree  of 
variability  was  present  on  many  of  the  variables,  across 
aircraft,  subjects,  freezes,  and  scenarios.  Possible 
sources  of  these  variations  will  be  examined  more 
closely. 


Analysis  of  Freeze  Number 

An  analysis  was  conducted  to  ascertain  it  there  was 
a  difference  between  the  first  and  second  freezes  in  the 
subjects’  ability  to  correctly  identify  what  was  hap¬ 
pening  in  the  scenarios.  The  first  freeze  always  oc¬ 
curred  two  minutes  before  the  OE  and  the  second 
freeze  always  occurred  at  the  time  of  the  error.  A 
multivariate  test  was  performed  on  the  accuracy  of 
subjects’  responses  across  the  queries  to  examine  dif¬ 
ferences  between  these  two  freeze  times.  (All  mea¬ 
sures,  expressed  as  percent  correct,  were  subjected  to 
an  arcsine  transformation  prior  to  analysis  to  meet  the 
conditions  of  ANOVA.)  The  MANOVA  was  not 
significant,  F(13,98)  =  1.554,  p>.05.  Therefore,  sub¬ 
jects’  recall  of  the  situation  was  not  significantly 
different  as  a  function  of  the  presence  of  an  OE. 

An  analysis  was  also  conducted  to  determine  whether 
subjects  reported  a  different  level  of  subjective  work¬ 
load  between  the  two  freezes,  as  there  is  a  concern  that 
higher  workload  may  be  associated  with  the  occur¬ 
rence  of  OEs.  The  NASA-TLX  ratings  were  weighted 
based  on  each  subject’s  rankings  and  a  combined 
workload  score  was  calculated.  An  ANOVA  was  con¬ 
ducted  on  the  combined  workload  ratings  to  test  for 
differences  between  the  two  freezes.  The  overall  NASA- 
TLX  workload  rating  was  significantly  higher  at  the 
time  of  the  second  freeze  (during  the  OE),  F(l,109)  = 
24.08,  p<.001,  as  shown  in  Figure  2. 

To  investigate  further,  ANOVAs  were  performed 
on  each  of  the  subscale  ratings  (performance,  tempo¬ 
ral  demand,  frustration,  mental  demand,  effort,  and 
physical  demand),  revealing  that  ratings  were  signifi¬ 
cantly  higher  at  the  second  freeze  for  all  of  the  subscales 
(see  Table  3)  except  for  physical  demand,  which  is  as 
would  be  expected.  This  supports  the  contention  that 
higher  workload  is  associated  with  OEs;  however,  it  is 
unclear  whether  higher  workload  caused  the  error,  or 
whether  the  higher  workload  ratings  were  the  result  of 
the  error  (i.e. ,  higher  ratings  on  the  subscales  of  stress, 
frustration,  demand,  effort,  and  performance  could  have 
been  an  outcome  of  the  fact  that  an  error  occurred). 

Analysis  of  Workload  Impact 

Subjective  workload.  Since  subjective  workload 
was  higher  at  the  time  of  the  OE,  further  analysis  was 
conducted  to  determine  if  there  was  a  direct  relationship 
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Figure  2.  Mean  Workload  Ratings  at  Each  Freeze 


Table  3.  ANOVAs:  Impact  of  Freeze  Number  on 
NASA-  TLX  Rating  and  Subscale  Scores 


Variable 

df 

F 

_ P 

NASA-TLX  (overall) 

1,  109 

24.079 

.001 

•k-k 

-  Performance 

1, 109 

41.705 

.001 

ic-k 

-  Temporal 

1, 109 

9.580 

.002 

** 

-  Frustration 

1, 109 

30.094 

.001 

ic-k 

-  Mental 

1,  109 

5.722 

.018 

★ 

-  Effort 

1,  109 

5.231 

.024 

★ 

-  Physical 

1.  109 

2.193 

.142 

*  significant  at  a  =  .05  levei 
**  significant  at  a  =  .01  ievei 
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Table  4.  Regressions:  Relationship  between  NASA-  TLX  Rating  and  Awareness 
of  the  Situation 


Variable _ _ _ _ df _ F 


Distance  error  (miles) 

.004 

1,  109 

.475 

.492 

Aircraft  reported  (%  correct) 

.064 

1,  109 

7.418 

.008 

Control  level  (%  correct) 

.006 

1.  109 

.674 

.413 

Call  sign:  alphabetic  (%  correct) 

.017 

1,  109 

1.882 

.173 

Call  sign:  numeric  (%  correct) 

.000 

1,  109 

.039 

.843 

Altitude  (%  correct) 

.001 

1,  109 

.120 

.730 

Change  in  altitude  (%  correct) 

.015 

1,  109 

1.715 

.193 

Speed  (%  correct) 

.018 

1,  109 

2.049 

.155 

Heading  (%  correct) 

.048 

1,  109 

5.484 

.021 

Turn  (%  correct) 

.018 

1,  109 

1.956 

.165 

Separation  problems  (%  correct) 

.023 

1,  109 

2.618 

.109 

Transition  to  next  sector  (%  correct) 

.022 

1,  109 

2.495 

.117 

Assigned  clearances  comolete  (%  correct) 

.065 

1.  109 

7.573 

.007 

-  all  measures  expressed  In  percentages  were  subjected  to  an  arcsine  transformation  prior 
to  analysis 

*  significant  at  a  =  .05  level 
**  significant  at  a  =  .01  level 


Table  5.  ANOVAs:  Relationship  between 

NASA- 

TLX  Rating 

and 

Awareness  of  the  Situation 

Variable 

df 

F 

D 

Assigned  clearance  correct  (y/n) 

1,79 

5.705 

.019 

Assigned  clearance  conformance  (y/n) 

1,79 

.388 

.535 

Weather  impact  (v/n) 

1.  109 

2.807 

.097 

*  significant  at  a  =  .05  level 
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between  the  NASA-TLX  combined  rating  and  sub¬ 
jects'  accuracy  in  their  awareness  of  the  situation. 
Regressions  were  performed  on  the  subjects'  scores  (in 
terms  of  percentage  correct)  in  comparison  to  their 
workload  ratings  at  the  time,  shown  in  Table  4.  Those 
variables  scored  as  correct  or  incorrect  were  subjected 
to  an  analysis  of  variance  to  determine  whether  work¬ 
load  ratings  were  associated  with  accuracy  of  recall, 
shown  in  Table  5.  A  MAN  OVA  showed  a  significant 
relationship  between  these  variables  and  the  NASA- 
TLX  score,  F  (14,96)  =  1.93,  p  <.05.  (Assignment 
correct  and  assignment  conformance  were  not  included 
in  the  multivariate  analysis  as  they  would  significantly 
reduce  the  degrees  of  freedom  in  the  test.  These  two 
questions  had  a  lower  sample  size  because  they  were 
only  asked  for  aircraft  that  had  not  yet  completed  their 
assigned  clearances.) 

The  regressions  revealed  that  subjects  reported  sig¬ 
nificantly  fewer  aircraft  present  (Figure  3)  and  identi¬ 
fied  aircraft  heading  correctly  significantly  less  often 


NASA*TLX 

Figure  3.  Impact  of  Subjective  Workload 
on  Aircraft  Reported 


for  those  aircraft  (Figure  4)  as  subjective  workload 
level  increased.  In  addition,  with  increasing  levels  of 
workload,  subjects  were  significantly  less  likely  to 
correctly  identify  whether  aircraft  had  completed  their 
assigned  clearances  (Figure  5)  and  were  significantly 
less  likely  to  identify  whether  an  aircraft  had  received 
its  assigned  clearance  correctly  (Figure  6). 

Number  of  aircraft.  An  objective  measure  of 
taskload  was  also  examined  to  determine  its  impact  on 
subject  awareness  and  perceived  workload.  The  num¬ 
ber  of  aircraft  present  at  the  time  of  each  freeze  was 
calculated  and  regressions  performed  to  examine  its 
relationship  to  accuracy  on  each  question  (in  terms  of 
percentage  correct),  shown  in  Table  6.  Those  vari¬ 
ables  scored  as  correct  or  incorrect  were  subjected  to 
an  analysis  of  variance  to  determine  whether  number 
of  aircraft  was  related  to  accuracy,  shown  in  Table  7. 
A  MAN OVA  showed  a  significant  relationship  be¬ 
tween  these  variables  and  the  number  of  aircraft 
present,  F  (14,97)  =  9.50,  p  <.001. 


Figure  4.  Impact  of  Subjective  Workload 
on  Awareness  of  Aircraft  Heading 
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NASA-TLX 

Figure  5.  Impact  of  Subjective  Workload  on 
Awareness  of  Aircraft  with  Clearances 
Not  Completed 


100 


^  90 
^  80 


Correct  Incorrect 

Figure  6.  Relationship  Between  Subjective  Workload  Rating  and  Awareness 
of  Correct  Receipt  of  Assigned  Clearance 
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As  the  number  of  aircraft  increased,  the  percentage 
of  aircraft  that  subjects  reported  as  present  signifi- 
candy  decreased  (Figure  7).  For  those  aircraft  that 
subjects  did  report,  subjects  were  also  significantly 
less  accurate  on  most  of  the  other  factors  about  those 
aircraft  as  the  number  of  aircraft  present  increased. 
They  were  significantly  more  erroneous  in  their  aware¬ 
ness  of  the  location  of  the  aircraft  (Figure  8),  and 
correct  less  frequently  regarding  the  aircrafts*  control 
level  (Figure  9),  both  the  alphabetic  portion  and 
numeric  portion  of  the  call  sign  (Figures  10  and  1 1), 


altitude  (Figure  12),  change  in  altitude  (Figure  13), 
airspeed  (Figure  14),  and  heading  (Figure  15)  as  the 
number  of  aircraft  increased.  They  were  also  correct 
significantly  less  frequently  in  their  awareness  of  which 
aircraft  would  transition  out  of  the  sector  in  the  next 
two  minutes  (Figure  16),  which  aircraft  had  completed 
their  assigned  clearances  (Figure  17),  and  if  weather 
would  be  a  factor  (Figure  18)  as  the  number  of  aircraft 
increased.  Interestingly,  the  number  of  aircraft  present 
did  not  affect  subjects*  accuracy  in  reporting  which 
aircraft  had  a  potential  or  current  separation  problem. 


Table  6.  Regressions:  Relationship  between  Number  of  Aircraft  and  Awareness 
of  Situation 


Variable 

,2 

df 

F 

D 

Distance  error  (miles) 

.052 

1,  110 

6.091 

.015 

ir 

Aircraft  reported  (%  correct) 

.331 

1.  110 

54.409 

.001 

*★ 

Control  level  (%  correct) 

.108 

1,  110 

13.324 

.001 

** 

Call  sign:  alphabetic  (%  correct) 

.075 

1,  110 

8.945 

.003 

** 

Call  sign:  numeric  (%  correct) 

.103 

1,  110 

12.619 

.001 

** 

Altitude  (%  correct) 

.080 

1,  lio 

9.556 

.003 

** 

Change  in  altitude  (%  correct) 

.057 

1,  110 

6.660 

.011 

* 

Speed  (%  correct) 

.150 

1,  110 

19.400 

.001 

Heading  (%  correct) 

.037 

1,  110 

4.187 

.043 

* 

Turn  (%  correct) 

.006 

1,  110 

.695 

.406 

Separation  problems  (%  correct) 

.003 

1,  110 

.386 

.536 

Transition  to  next  sector  (%  correct) 

.128 

1,  110 

16.200 

.001 

** 

Assianed  clearances  complete  (%  correct) 

.072 

1.  110 

8.479 

.004 

** 

-  all  measures  expressed  in  percentages  were  subjected  to  an  arcsine  transformation  prior 
to  analysis 

*  significant  at  a=  .05  level 
**  significant  ata  =  .01  level 


Table  7.  ANOVAs:  Relationship  between  Number  of  Aircraft  and 

Awareness  of  Situation 

Variable 

df 

F 

_  P 

Assigned  clearance  correct  (y/n) 

1,80 

.003 

.960 

Assigned  clearance  conformance  (y/n) 

1,80 

2.789 

.099 

Weather  impact  (v/n) 

1. 110 

8.898 

.004 

*  significant  at  a-  .05  level 
**  significant  ata=:  .01  level 
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Number  of  Aircraft 

Figure  7.  Impact  of  Number  of  Aircraft  on 
Aircraft  Reported 


Number  of  Aircraft 

Figure  8.  Impact  of  Number  of  Aircraft  on 
Aircraft  Placement  Accuracy 


Number  of  Aircraft 

Figure  9.  Impact  of  Number  of  Aircraft  on 
Awareness  of  Aircraft  Control  Level 


Number  of  Aircraft 

Figure  10.  Impact  of  Number  of  Aircraft  on 
Awareness  of  Aircraft  Callsign  (Alphabetic) 
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Numbw  of  Alreratt  Number  of  Aircraft 

Figure  1 1 .  Impact  of  Number  of  Aircraft  on  Figure  1 2.  Impact  of  Number  of  Aircraft  on 

Awareness  of  Aircraft  Callsign  (Numeric)  Awareness  of  Aircraft  Altitude 


Number  of  Abcraft  Number  of  Aircraft 

Figure  13.  Impact  of  Number  of  Aircraft  on  Figure  14.  Impact  of  Number  of  Aircraft  on 

Awareness  of  Aircraft  Change  in  Altitude  Awareness  of  Aircraft  Speed 
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%  Correct  ^  Correct 


Number  of  Aircraft 

Figure  15.  Impact  of  Number  of  Aircraft  on 
Awareness  of  Aircraft  Heading 


Number  of  Aircraft 

Figure  17.  Impact  of  Number  of  Aircraft  on 
Awareness  of  Aircraft  Assignment  Completion 


Number  of  Aircraft 

Figure  16.  Impact  of  Number  of  Aircraft  on 
Awareness  of  Aircraft  Transitioning  Out  of  Sector 
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Knowledge  of  Weather 

Figure  18.  Impact  of  Number  of  Aircraft  on 
Awareness  of  Weather 
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Correlation  between  measures.  An  analysis  was 
also  conducted  to  determine  the  relationship  between 
the  number  of  aircraft  present,  as  an  objective  measure 
of  taskload,  and  subject’s  reported  subjective  work¬ 
load.  A  regression  showed  a  significant  relationship 
between  these  two  measures  of  load,  F(l,109)  =  6.45, 
p  =  .013,  as  shown  in  Figure  19.  It  appears  that  the 
number  of  aircraft  present  in  a  scenario  is  a  significant 
driver  of  perceived  workload  that  negatively  impacts 
subjects’  ability  to  keep  an  accurate  mental  picture  of 
the  situation,  independent  of  any  other  factors  that 
might  also  drive  subjective  workload.  An  increase  in 
the  number  of  aircraft  present  not  only  was  related  to 
a  tendency  to  attend  to  fewer  aircraft,  but  also  to  the 
tendency  to  know  significantly  less  about  these  air¬ 
craft.  This  load-related  shedding  of  information  at¬ 
tended  to  appears  to  reflect  some  prioritization  of 
tasks,  as  awareness  of  those  aircraft  with  current  or 
potential  separation  problems  was  not  significantly 
impacted  by  workload. 

Analysis  of  Operational  Errors 

A  closer  examination  of  the  nature  of  the  OEs 
included  in  this  study  was  made.  Table  8  provides  a 
summary  of  each  error.  Each  error  was  classified  in 
terms  of  the  SA  Error  Taxonomy  (Endsley,  1995a). 
Classifications  were  made  by  two  independent  raters 
based  on  a  description  of  each  OE  contained  in  the 
Final  Operational  Error/Deviation  Report  completed 
by  internal  FAA  quality  assurance  investigators  after 
the  OE,  an  analysis  of  each  OE  from  the  SATORI  re¬ 
creation,  and  verbal  comments  made  by  the  subjects 
in  this  study. 

Of  the  14  errors  investigated,  5  clearly  involved 
task  distractions  in  which  the  controller  was  dis¬ 
tracted  by  the  need  to  attend  to  other  aircraft  in  the 
sector.  Three  involved  the  misperception  of  some 
information  (due  to  expectations,  workload,  or  task 
distraction).  Two  OEs  involved  memory  loss  (associated, 
with  task  distraction  and  high  workload)  in  which  the 
controller  forgot  about  an  aircraft  or  a  previous  action. 
Two  OEs  involved  an  over-reliance  on  defaults,  expect¬ 
ing  aircraft  to  behave  as  they  usually  do.  Four  involved 
problems  with  inadequate  projection  of  the  dynamics  of 
the  aircraft  to  anticipate  separation  problems.  High 
workload  was  specifically  cited  as  a  problem  in  one  OE. 


Figure  19.  Relationship  Between  Number  of 
Aircraft  and  Subjective  Workload  Ratings 


Because  controllers  routinely  must  cope  with  com¬ 
peting  task  demands  and  high  workload  situations,  an 
analysis  was  made  to  determine  whether  the  situations 
in  which  these  OEs  occurred  were  intrinsically  likely 
to  induce  errors.  Each  of  the  investigated  errors  is 
listed  in  Table  9,  along  with  a  listing  of  the  number  of 
aircraft  in  the  sector  at  the  time  of  the  OE  and  a  rating 
of  the  complexity  of  the  scenario  (as  reported  in  the 
Final  Operational  Error/Deviation  Report).  (This 
information  was  not  included  in  the  reports  for  Sce¬ 
narios  6,  12  and  13.)  Scenarios  1,  2,  and  14  had  the 
highest  level  of  complexity  and  highest  number  of 
aircraft  in  the  sector  at  the  time  of  the  error,  followed 
by  scenarios  9  and  11.  Scenarios  3,  4,  7,  and  10  had 
lower  than  average  complexity  and  number  of  aircraft 
at  the  time  of  the  error.  This  agrees  with  previous 
findings  that  many  OEs  occur  during  low  workload 
situations. 

Table  9  also  lists  whether  the  controller  making  the 
OE  was  aware  that  the  OE  was  occurring  at  the  time. 
(This  information  was  not  available  for  two  of  the 
errors.)  In  4  out  of  the  remaining  1 2  OEs,  the  control¬ 
ler  was  aware  than  an  error  was  building  but  was  not 
able  to  avoid  it.  In  the  remaining  eight,  the  controller 
was  not  aware  that  an  OE  had  occurred. 
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Table  8.  Summary  of  Operational  Errors  Investigated 
Scenario  Description  of  Error 


Description  of  Error 

SA  Error  Type 

Clearance  to  wrong  aircraft 

Level  1  -  misperception 

Level  1  -  failure  to  monitor 
(task  distraction) 

Descended  aircraft  into  other  aircraft 

Level  2  -  inadequate  mental  model 
Level  3  -  lack  of  projection 

difference  In  climb/closure  rates  of 
two  aircraft 

Level  3  -  lack  of  projection 

difference  in  climb/closure  rates  of 
two  aircraft 

Level  3  -  lack  of  projection 

Expected  aircraft  to  descend  faster 
than  it  did 

Level  2  -  over-reliance  on  defaults 

Readback  error 

Level  1  -  misperception 
(expectations) 

Delay  in  turning  aircraft  to  accomodate 
slow  descent  of  other  aircraft 

Level  3  -  lack  of  projection 

Forgot  aircraft  -  provided  inadequate 

Level  1  -  memory  loss 

clearance 

task  distraction 

Did  not  separate  aircraft 

Level  1  -  failure  to  monitor 
task  distraction 

Readback  error  -  aircraft  at  different 

Level  1  -  misperception 

altitude 

task  distraction 

Gave  wrong  heading  command 

Didn’t  monitor  compliance 

Level  1  -  failure  to  monitor 

Descended  aircraft  into  other  aircraft 

Level  1  -  failure  to  monitor 

Level  2  -  over-reliance  on  defaults 

Climbed  aircraft  into  other  aircraft 
didn’t  judge  separation 

Level  1  -  task  distraction 

Issued  clearance  to  aircraft  off  frequency 

Level  1  -  memory  loss  -  workload 

Lost  track  of  slow  aircraft  climb  rate 

Level  1  -  failure  to  monitor-  workload 

Confused  two  aircraft 
Attending  to  other 
aircraft 


Table  9.  Subject  Awareness  of  Error 
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As  a  comparison,  the  responses  of  the  subjects  in 
this  study  were  examined  to  determine  whether  they 
were  aware  that  separation  errors  were  developing. 
The  percentage  of  the  four  subjects  viewing  each 
scenario  who  reported  the  existence  of  the  two  aircraft 
involved  in  the  OE  at  each  of  the  two  freezes  is  listed 
in  Table  9.  While  for  many  of  the  scenarios  both 
aircraft  were  reported  by  all  four  subjects,  at  least  one 
of  the  involved  aircraft  was  not  reported  at  all  by  at 
least  one  subject  in  approximately  one-half  of  the 
scenarios  at  the  first  freeze  and  approximately  one- 
third  of  the  scenarios  at  the  second  freeze. 

In  their  response  to  the  question  regarding  which 
aircraft  had  lost,  or  would  lose  separation  in  the  next 
two  minutes,  in  only  four  of  the  scenarios  did  even 
one  of  the  four  subjects  list  the  two  aircraft  involved 
in  the  OE  at  the  freeze,  which  occurred  two  minutes 
before  the  OE.  At  the  time  of  the  second  freeze,  when 
the  OE  occurred,  at  least  one  of  the  four  subjects  did 
not  identify  the  separation  error  in  eight  of  the  four¬ 
teen  scenarios.  In  three  of  these  cases,  they  reported 
both  aircraft  but  did  not  indicate  a  separation  prob¬ 
lem.  In  the  remaining  five  scenarios,  they  also  failed  to 
report  at  least  one  of  the  involved  aircraft,  indicating 
it  was  outside  of  their  focus. 

Based  on  this  analysis,  it  would  appear  that  the 
factors  causing  the  controller  to  make  an  error  in 
many  of  these  OEs  were  significant  enough  to  have 
been  a  problem  for  other  trained  controllers.  This 
result  should  be  viewed  with  caution,  however,  as  the 
method  employed  in  this  study  involved  passive  viewing 
of  the  scenarios,  instead  of  being  under  actual  opera¬ 
tional  control. 

CONCLUSIONS 

This  study  reveals  many  interesting  findings  on  the 
role  of  situation  awareness  and  workload  in  opera¬ 
tional  errors.  Significant  deficiencies  in  the  ongoing 
situation  awareness  of  the  subjects  were  present  in  this 
study.  They  had  a  fairly  low  ability  to  report  on  the 
existence  of  many  aircraft,  or  accurately  recall  their 
location  or  many  of  their  parameters.  Their  accuracy 
was  significantly  impacted  by  the  number  of  aircraft 
present  in  the  scenario  and,  to  a  lesser  degree,  by 
perceived  workload.  After  the  number  of  aircraft 


present  exceeded  approximately  eight  to  ten,  the  abil¬ 
ity  of  subjects  to  report  on  each  aircraft  declined  quite 
rapidly.  Even  for  those  aircraft  they  did  report  on, 
their  awareness  of  the  relevant  parameters  for  these 
aircraft  also  declined  when  there  were  more  than 
approximately  ten  aircraft  present.  In  the  face  of  a 
high  number  of  aircraft,  subjects  tended  to  attempt  to 
maintain  their  awareness  of  aircraft  separation;  how¬ 
ever,  this  was  still  less  than  perfect  (86.2%  correct  on 
average).  Other  tasks,  such  as  follow  through  on 
clearances  given  to  aircraft,  also  appeared  to  suffer 
under  increases  in  the  number  of  aircraft  and  per¬ 
ceived  workload. 

While  it  is  difficult  to  say  that  controllers  need  to 
be  able  to  remember  aircraft  parameters  as  long  as  they 
know  about  aircraft  separation,  the  pattern  of  atten¬ 
tion  represented  by  the  accuracy  scores  in  this  study 
are  indicative  of  many  of  the  OEs  that  occur.  Readback 
errors,  for  instance,  may  be  directly  related  to  the 
tendency  for  subjects  in  this  study  to  have  a  fairly  low 
awareness  of  whether  aircraft  given  clearances  had 
received  the  clearance  correctly.  (Subjects  were  only 
correct  in  23.2%  of  the  cases  in  knowing  if  an  aircraft 
had  completed  its  assigned  clearance,  and  of  these,  in 
only  74.4%  of  the  cases  did  they  know  if  the  aircraft 
had  received  its  assigned  clearance  correctly.)  Prob¬ 
lems  in  reporting  the  numeric  portion  of  callsigns  is 
also  reflective  of  OE  error  patterns.  Many  of  the  OEs 
included  in  this  study  involved  over-reliance  on  ex¬ 
pectations  about  how  fast  aircraft  would  travel  or 
should  ascend  or  descend.  Low  accuracy  in  reporting 
on  the  speed  and  heading  of  aircraft  and  being  able  to 
report  whether  aircraft  were  turning,  ascending,  or  de¬ 
scending  are  most  likely  indicative  of  this  type  of  error. 

An  important  issue  is  why  these  SA  problems  occur. 
It  is  a  mistake  to  infer  that  these  subjects  (or  the 
controllers  involved  in  the  errors)  were  simply  inat¬ 
tentive.  Everyone  has  a  limited  amount  of  attention  to 
distribute  in  any  situation  (Wickens,  1992).  In  com¬ 
plex  activities,  such  as  air  traffic  control,  there  is  a 
great  deal  of  information  to  process.  The  pattern  of 
errors  in  this  study  suggests  that,  even  if  subjects  made 
as  much  use  of  their  attention  as  possible  in  keeping 
up  with  the  scenarios,  they  may  have  had  to  limit 
attention  to  some  information  in  order  to  keep  up 
with  the  need  to  ensure  that  all  aircraft  were  separated. 
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Probabilistically,  this  strategy  is  effective  much  of  the 
time,  but  is  likely  to  produce  occasional  errors  of  the 
type  described  here. 

Information  regarding  the  role  of  workload  in  the 
occurrence  of  OEs  is  also  present  here.  Subjective 
workload,  as  measured  by  NASA-TLX,  was  signifi¬ 
cantly  higher  at  the  time  of  the  operational  error  than 
it  was  two  minutes  prior  in  the  scenarios.  (The  num¬ 
ber  of  aircraft  was  not  higher,  however.)  While  the 
NASA-TLX  score  was  correlated  with  the  number  of 
aircraft  present,  the  number  of  aircraft  present  was 
more  closely  related  to  decreases  in  subjects’  aware¬ 
ness  of  the  situation  (beyond  a  certain  number  of 
aircraft).  (What  other  factors  are  included  by  subjects 
in  their  subjective  workload  ratings  is  unknown.) 

It  is  important  to  note,  however,  that  low  and 
moderate  levels  of  SA  were  also  present  on  many 
variables  even  when  the  number  of  aircraft  was  rela¬ 
tively  low.  This  finding  concurs  with  other  studies 
that  have  found  that  SA  and  workload  can  operate 
independently  for  various  reasons  (Endsley,  1993). 
OEs  were  also  found  to  occur  under  high-,  moderate-, 
and  low-load  conditions  (as  indicated  by  the  num¬ 
ber  of  aircraft  present).  Thus,  while  workload  and 
in  many  of  these  cases,  momentary  task  distrac¬ 
tion,  can  be  associated  with  loss  of  SA  and  OEs, 
errors  can  also  occur  for  other  reasons  at  lower 
levels  of  workload. 

One  limitation  of  this  study  is  that  it  involved 
subjects  who,  although  current,  experienced  control¬ 
lers,  passively  viewed  the  ATC  scenario  re-creations. 
While  this  procedure  provided  a  great  deal  of  insight 
into  factors  affecting  actual  OEs  (which  can  be  diffi¬ 
cult  to  produce  under  simulated  conditions),  one  can 
speculate  as  to  whether  their  situation  recall  accuracy 
and  subjective  workload  ratings  are  the  same  as  they 
would  be  for  controllers  actively  working  the  same 
scenarios.  While  it  is  probably  difficult  to  stipulate 
that  the  absolute  levels  of  SA  and  workload  reported 
would  be  the  same,  the  general  patterns  presented 
here  are  probably  valid  as  reflections  of  subjects’ 
attention  distribution. 

It  is  somewhat  likely,  however,  that  levels  of  SA 
may  be  lower  under  passive  viewing  conditions,  rather 


than  active  decision  making,  such  as  has  been  demon¬ 
strated  under  higher  levels  of  automation  in  recent 
research  (Endsley  &  Kiris,  1995).  In  this  light,  diffi¬ 
culties  in  accurately  identifying  aircraft  separation 
problems  shown  by  subjects  in  this  study  may  be  at 
least  partially  reflective  of  the  difficulties  associated 
with  passive  monitoring.  This  possibility  needs  to  be 
seriously  investigated  with  regard  to  systems  being 
developed  for  automating  future  air  traffic  control. 

As  a  basis  of  comparison,  Mogford  and  Tansley 
(1991),  using  a  procedure  similar  to  SAGAT  during 
actual  simulations  of  air  traffic  control  with  controller 
trainees,  found  that  subjects  were  able  to  report  air¬ 
craft  position  with  86%  accuracy,  heading  82%,  alti¬ 
tude  73%,  callsign  55%  and  speed  53%.  In  comparison 
with  the  data  we  obtained  during  passive  viewing,  the 
attention  devoted  to  each  type  of  information  is 
similarly  distributed;  however,  the  Mogford  and 
Tansley  study  showed  much  higher  levels  of  SA,  even 
though  they  studied  controller  trainees  who  would  be 
theorized  to  have  lower  SA  than  Full  Performance 
Level  controllers.  This  supports  the  contention  that 
SA  may  be  compromised  under  passive  viewing,  which 
should  be  a  significant  concern  for  automation  sys¬ 
tems  designed  to  place  the  controller  in  the  role  of 
passive  monitor. 

In  conclusion,  this  study  may  indicate  several 
sources  leading  to  operational  errors.  The  degree  to 
which  these  results  are  generalizable  to  controllers 
involved  in  actively  controlling  air  traffic  needs  to  be 
investigated,  since  the  method  employed  in  this  study 
involved  passive  viewing  of  the  scenario,  instead  of 
actual  operational  control  of  the  aircraft.  Comparable 
data  need  to  be  examined  during  simulations  of  air 
traffic  control  to  verify  similar  attention  allocation  to 
situation  variables  and  responses  to  workload.  At  the 
very  least,  issues  regarding  SA  demonstrated  in  this 
study  may  be  indicative  of  problems  that  can  be 
expected  if  air  traffic  control  ever  becomes  highly 
automated,  relegating  the  controller  to  a  monitor. 
Alternate  automation  designs  that  keep  the  controller 
in  the  active  decision  making  loop  need  to  be  explored 
to  prevent  such  an  outcome. 
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